Song selection using a heart rate change and a facial expression monitored with a camera

ABSTRACT

A method, a computer program product, and a computer system for selecting songs using a heart rate change, a blood pressure change, and a facial expression of a listener. A computer receives a song from a music source selected by a listener and determines whether the song is appropriate. The computer starts to play the song, in response to determining that the song is appropriate. A camera connected to the computer detects a facial expression of the listener. The computer determines whether the facial expression is positive. In response to determining that the facial expression is positive, the computer the song in full. The camera connected to the computer detects a heart rate change and a blood pressure change of the listener. The computer tags the song with updated information.

BACKGROUND OF THE INVENTION

The present invention relates generally to computer implementedanalytics for music selection, and more particularly to song selectionusing a heart rate change and a facial expression of a listenermonitored with a camera such as a webcam.

The introduction of more portable and accessible music has led manypeople to incorporating music into their work environments. Music canblock out noisy distractions of an office and can also keep a listenermotivated, focused, and released from stress. The key to optimumproductivity is finding music that can best keeps the listenermotivated, focused, and relieved from stress. However, to findappropriate music for optimizing productivity is time consuming; thespent time for finding the appropriate music may negate the benefits themusic can provide.

SUMMARY

In one aspect, a method for selecting songs using a heart rate change, ablood pressure change, and a facial expression of a listener isprovided. The method is implemented by a computer. The method comprisesreceiving a song from a music source selected by a listener; determiningwhether the song is appropriate, based on clock information, a currentheart rate and a current blood pressure detected by a camera connectedto the computer, and a target of a heart rate and a blood pressure forthe listener; starting to play the song, in response to determining thatthe song is appropriate; detecting, by the camera connected to thecomputer, a facial expression of the listener; determining whether thefacial expression is positive; playing the song in full, in response todetermining that the facial expression is positive; detecting, by thecamera connected to the computer, a heart rate change and a bloodpressure change of the listener; and tagging the song with updatedinformation.

In another aspect, a computer program product for selecting songs usinga heart rate change, a blood pressure change, and a facial expression ofa listener is provided. The computer program product comprises acomputer readable storage medium having program code embodied therewith.The program code is executable to: receive a song from a music sourceselected by a listener; determine whether the song is appropriate, basedon clock information, a current heart rate and a current blood pressuredetected by a camera connected to the computer, and a target of a heartrate and a blood pressure for the listener; start to play the song, inresponse to determining that the song is appropriate; detect, by thecamera connected to the computer, a facial expression of the listener;determine whether the facial expression is positive; play the song infull, in response to determining that the facial expression is positive;detect a heart rate change and a blood pressure change of the listener;and tag the song with updated information.

In yet another aspect, a computer system for selecting songs using aheart rate change, a blood pressure change, and a facial expression of alistener is provided. The computer system comprises one or moreprocessors, one or more computer readable tangible storage devices, andprogram instructions stored on at least one of the one or more computerreadable tangible storage devices for execution by at least one of theone or more processors. The program instructions are executable toreceive a song from a music source selected by a listener. The programinstructions are executable to determine whether the song isappropriate, based on clock information, a current heart rate and acurrent blood pressure detected by a camera connected to the computer,and a target of a heart rate and a blood pressure for the listener. Theprogram instructions are executable to start to play the song inresponse to determining that the song is appropriate. The programinstructions are executable to detect, by the camera connected to thecomputer, a facial expression of the listener. The program instructionsare executable to determine whether the facial expression is positive.The program instructions are executable to play the song in full inresponse to determining that the facial expression is positive. Theprogram instructions are executable to detect a heart rate change and ablood pressure change of the listener. The program instructions areexecutable to tag the song with updated information.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a flowchart showing operational steps for selecting songsusing a heart rate change, a blood pressure change, and a facialexpression of a listener, in accordance with one embodiment of thepresent invention.

FIG. 2 is a flowchart showing operational steps for tagging a song withupdated metadata, in accordance with one embodiment of the presentinvention.

FIG. 3 is a flowchart showing operational steps for determining whethera song is appropriate, in accordance with one embodiment of the presentinvention.

FIG. 4 is a diagram illustrating components of a computer device hostinga program for selecting a song using a heart rate change, a bloodpressure change, and a face expression of a listener, in accordance withone embodiment of the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention discloses a system for selectingsongs using a heart rate change, a blood pressure, and a facialexpression of a listener. Using a camera such as a webcam on a laptop,the system measures the heart rate change of the listener in aworkplace, and dynamically choose songs that help the listener achieveand maintain the perfect work focus balance. By detecting the facialexpression of the listener with the camera, the system also detectswhether the listener wishes to skip a song. The system may pause thesong if the listener leaves the workplace. The system allows thelistener to stay focused on work without having to switch the focus tomanaging the music.

The system uses the combination of facial expression recognition andimage processing to determine the heart rate and the blood pressure;therefore, the listener does not need to leave his/her work screens tochange or skip songs. By certain expressions, the listener can dictate asong to be skipped or validate that they enjoy the particular type ofsong that is playing. The system can even pause the music when alistener leaves his/her desk (for example, he/she needs to leave quicklyfor a meeting), saving his/her time scrambling to find the windowplaying music from his/her work windows. At the end of a work session,the system can capture listener's feedback about when he/she is the mostfocused; therefore, the system can use the feedback to further tailormusic choices and expressions to the listener.

Music listening is an increasing occurrence in workplaces and, throughthe system, the natural trend can be harnessed to improve overall workproductivity while potentially promoting happier work environments. Thesystem can also be used in areas other than workplaces, for example ingaming. Many gamers prefer to listen to their own custom music over thegame music. By measuring the gamer's heart rate, the system can providesongs tailored to the gamer. When the gamer's heart rate is high, if thegamer is playing a fast paced action game, more intense songs to matchthe mood can be played; conversely, if the gamer is more relaxed (i.e.,if the gamer is playing an adventure game), the system will play a morerelaxed song to match the mood. The heart rate balancing song selectionmethod may also be applicable either by making a frustrated gamer morerelaxed or improve a boring game.

FIG. 1 is flowchart 100 showing operational steps for selecting songsusing a heart rate change, a blood pressure change, and a facialexpression of a listener, in accordance with one embodiment of thepresent invention. At step 101, a computer (showing in FIG. 4) receivesa song provided by a music source, a playlist or a streaming site.First, the song is provided by any playlist or streaming site a listenerchooses. By using a large playlist provided by the listener or astreaming site which tailors music to the listener's listeningpreference, the system not only be ensured to play the most appropriatesong to match their target work rate but also plays songs that the userenjoys putting them in a more positive mood. At step 102, the computeridentifies the song. From the song details, the computer retrievesprevious metadata on a heart rate change and a feedback of the listener,if the song has been played. If no metadata has ever been captured, thecomputer may use a music analyzer to either find a song whose resultshave been captured or make standard decisions based on the heart ratechange or mood of the song.

At step 103, the computer determines whether the song has been analyzedbefore. In response to determining that the song has not been analyzedbefore (NO branch of step 103), the computer at step 104 determineswhether the song is appropriate for a current situation based on asimilar song that has been analyzed. After step 104 or in response todetermining that has been analyzed before (YES branch of step 103), thecomputer at step 105 determines whether the song is appropriate for acurrent situation, based on clock information, a current heart rate anda current blood pressure detected by a camera connected to the computer,and a target of a heart rate and a blood pressure for the listener.

In response to determining that the song is appropriate for a currentsituation (YES branch of step 105), the computer at step 106 detects afacial expression of the listener. The facial expression provides animportant behavioral measure for studding emotion of the listener. Thefacial expression is detected with a camera such as a webcam on thecomputer. At step 107, the computer determines whether the facialexpression is positive. The computer analyzes the facial expressiondetected at step 106 and determines whether the listener has a positivemood. In response to determining that the facial expression is positive(YES branch of step 107), the computer at step 109 plays the full song.

At step 110, the computer detects a heart rate change and a bloodpressure change of the listener. The heart rate change and the bloodpressure change show differences of a heart rate and a blood pressurebefore and after listening the song. The heart rate and the bloodpressure of the listener is detected by a camera such as a webcam.Existing open libraries for heart beat monitoring through webcams workbest when the listener is sitting still and looking at the screen. Thewebcam can detect the heart rate of the listener through Eulerian VideoMagnification. It can detect slight changes in the listener's foreheadand facial imagery to determine the heart rate. The process can also beused to detect Mayer's waves which can be used to monitor a bloodpressure as well.

In response to determining that the song is not appropriate for acurrent situation (NO branch of step 105) or in response to determiningthat the facial expression is not positive (NO branch of step 107), thecomputer at step 108 skips the song. The song will not be played. Afterstep 108 or step 110, the computer at step 111 tags the song withupdated metadata. Tagging the song will be discussed in detail in thenext paragraphs with reference to FIG. 2.

FIG. 2 is flowchart 200 showing operational steps for tagging the songwith updated metadata, in accordance with one embodiment of the presentinvention. At step 201, a computer receives metadata of the song. Themetadata includes a music analysis result, a heart rate change of thelistener, and a facial expression of listener. At step 202, the computerdetermines whether there is previous metadata of the song.

In response to determining that there is the previous metadata of thesong (YES branch of step 202), the computer at step 203 compares themetadata of the song with the previous date of the song. In response todetermining that there is no previous metadata of the song (NO branch ofstep 202), the computer at step 204 compares the metadata of the songwith previous date of a similar song.

After step 203 or step 204, the computer at step 205 analyzes themetadata received at step 201 and the previous metadata of the song orthe similar song. When the computer analyzes the song with the previousmetadata the song, the computer will determine whether the song workswell as the previous time and determine the reason. When the computeranalyzes the song with the previous metadata the similar song, thecomputer will determine whether a result of the song is similar to anexpected result and determine the reason. At step 206, the computerupdates the metadata of the song in a database.

FIG. 3 is flowchart 300 showing operational steps for determiningwhether a song is appropriate, in accordance with one embodiment of thepresent invention. At step 301, the computer receives a song provided bya playlist or a streaming site. At step 302, the computer receives clockinformation. The computer makes use of a clock to further optimize musicto the work day. For example, faster paced adrenaline producing musicmay be played in the morning when the listener needs an extra boost towake up; more relaxing soothing music may be played in the lateafternoon to help prevent the listener from being burnt out by stress.At step 303, the computer receives information of a current heart rateand a current blood pressure detected by a camera such as a webcam. Atstep 304, the computer receives information about a target of a heartrate and a blood pressure for the listener. The target of the heart rateand the blood pressure for each listener is derived first from thegender and age of the listener and then is personalized through trainingas the computer learns at which heart rate and blood pressure thelistener feels they are the most focused. The target of the heart rateand the blood pressure can be specified based on the time of day (forexample, walking hours, work time, lunch, or scheduled breaks).

At step 305, the computer determines whether the song is appropriate forthe target. When the song is provided from a queue of the playlist orthe streaming site, computer uses the clock information, the currentheart rate information, and the target of the heart rate and the bloodpressure which is unique to the listener, the computer compares theinformation received at steps 302, 303, and 304 with the metadatacollected for the queued song. In response to determining that the songis appropriate for the target (YES branch of step 305), at step 306, thecomputer plays the song. Otherwise, in response to determining that thesong is not appropriate for the target (NO branch of step 305), at step307, the computer skips the song.

FIG. 4 is a diagram illustrating components of components of computerdevice 400 hosting a program for selecting a song using a heart rate anda face expression, in accordance with one embodiment of the presentinvention. It should be appreciated that FIG. 4 provides only anillustration of one implementation and does not imply any limitationswith regard to the environment in which different embodiments may beimplemented.

Referring to FIG. 4, computer device 400 includes processor(s) 420,memory 410, and tangible storage device(s) 430. In FIG. 4,communications among the above-mentioned components of computer device400 are denoted by numeral 490. Memory 410 includes ROM(s) (Read OnlyMemory) 411, RAM(s) (Random Access Memory) 413, and cache(s) 415. One ormore operating systems 431 and one or more computer programs 433 resideon one or more computer readable tangible storage device(s) 430. The oneor more computer programs 433 include the program for selecting a songusing a heart rate change, a blood pressure change, and a faceexpression. Computer device 400 further includes I/O interface(s) 450.I/O interface(s) 450 allows for input and output of data with externaldevice(s) 460 that may be connected to computer device 400. Computerdevice 400 further includes network interface(s) 440 for communicationsbetween computer device 400 and a computer network.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device, such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network(LAN), a wide area network (WAN), and/or a wireless network. The networkmay comprise copper transmission cables, optical transmission fibers,wireless transmission, routers, firewalls, switches, gateway computersand/or edge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++, and conventionalprocedural programming languages, such as the “C” programming language,or similar programming languages. The computer readable programinstructions may execute entirely on the user's computer, partly on theuser's computer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer, or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider). In some embodiments,electronic circuitry including, for example, programmable logiccircuitry, field-programmable gate arrays (FPGA), or programmable logicarrays (PLA) may execute the computer readable program instructions byutilizing state information of the computer readable programinstructions to personalize the electronic circuitry in order to performaspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture, including instructions which implement aspectsof the function/act specified in the flowchart and/or block diagramblock or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus, or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the FIGs illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the FIGs. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

1-6. (canceled)
 7. A computer program product for selecting songs usinga heart rate change, a blood pressure change, and a facial expression ofa listener, the computer program product comprising a computer readablestorage medium having program code embodied therewith, the program codeexecutable to: receive, by a computer, a song from a music sourceselected by a listener; determine, by the computer, whether the song isappropriate, based on clock information, a current heart rate and acurrent blood pressure detected by a camera connected to the computer,and a target of a heart rate and a blood pressure for the listener;start, by the computer, to play the song, in response to determiningthat the song is appropriate; detect, by the camera connected to thecomputer, a facial expression of the listener; determine, by thecomputer, whether the facial expression is positive; play, by thecomputer, the song in full, in response to determining that the facialexpression is positive; detect, by the camera connected to the computer,a heart rate change and a blood pressure change of the listener; andtag, by the computer, the song with updated information.
 8. The computerprogram product of claim 7, further comprising the program codeexecutable to: receive, by the computer, the clock information; receive,by the computer, the current heart rate and the blood pressure of thelistener; receive, by the computer, the target of the heart rate and theblood pressure for the listener; and compare, by the computer, the clockinformation, the current heart rate and the blood pressure, and thetarget of the heart rate and the blood pressure with previous metadataof the song so as to determine whether the song is appropriate for thetarget.
 9. The computer program product of claim 7, further comprisingthe program code executable to: receive, by the computer, metadata ofthe song, the metadata including a music analysis result, the heart ratechange, the facial expression of the listener; determine, by thecomputer, whether there is previous metadata of the song; compare, bythe computer, the metadata of the song with the previous metadata of thesong, in response to determining that there is the previous metadata ofthe song; analyze, by the computer, the metadata of the song and theprevious metadata of the song; and update, by the computer, the metadataof the song in a database.
 10. The computer program product of claim 9,further comprising the program code executable to: compare, by thecomputer, the metadata of the song with previous metadata of a similarsong, in response to determining that there is no previous metadata ofthe song; analyze, by the computer, the metadata of the song and theprevious metadata of the similar song; and update, by the computer, themetadata of the song in the database.
 11. The computer program productof claim 7, further comprising the program code executable to: skip, bythe computer, the song, in response to determining that the song is notappropriate; and tag, by the computer, the song.
 12. The computerprogram product of claim 7, further comprising the program codeexecutable to: skip, by the computer, the song, in response todetermining that the facial expression is not positive; and tag, by thecomputer, the song.
 13. A computer system for selecting songs using aheart rate change, a blood pressure change, and a facial expression of alistener, the computer system comprising: one or more processors, one ormore computer readable tangible storage devices, and programinstructions stored on at least one of the one or more computer readabletangible storage devices for execution by at least one of the one ormore processors, the program instructions executable to: receive, by acomputer, a song from a music source selected by a listener; determine,by the computer, whether the song is appropriate, based on clockinformation, a current heart rate and a current blood pressure detectedby a camera connected to the computer, and a target of a heart rate anda blood pressure for the listener; start, by the computer, to play thesong, in response to determining that the song is appropriate; detect,by the camera connected to the computer, a facial expression of thelistener; determine, by the computer, whether the facial expression ispositive; play, by the computer, the song in full, in response todetermining that the facial expression is positive; detect, by thecamera connected to the computer, a heart rate change and a bloodpressure change of the listener; and tag, by the computer, the song withupdated information.
 14. The computer system of claim 13, furthercomprising the program instructions executable to: receive, by thecomputer, the clock information; receive, by the computer, the currentheart rate and the blood pressure of the listener; receive, by thecomputer, the target of the heart rate and the blood pressure for thelistener; and compare, by the computer, the clock information, thecurrent heart rate and the blood pressure, the target of the heart rateand the blood pressure with previous metadata so as to determine whetherthe song is appropriate for the target.
 15. The computer system of claim13, further comprising the program instructions executable to: receive,by the computer, metadata of the song, the metadata including a musicanalysis result, the heart rate change, the facial expression of thelistener; determine, by the computer, whether there is previous metadataof the song; compare, by the computer, the metadata of the song with theprevious metadata of the song, in response to determining that there isthe previous metadata of the song; analyze, by the computer, themetadata of the song and the previous metadata of the song; and update,by the computer, the metadata of the song in a database.
 16. Thecomputer system of claim 15, further comprising the program instructionsexecutable to: compare, by the computer, the metadata of the song withprevious metadata of a similar song, in response to determining thatthere is no previous metadata of the song; analyze, by the computer, themetadata of the song and the previous metadata of the similar song; andupdate, by the computer, the metadata of the song in the database. 17.The computer system of claim 13, further comprising the programinstructions executable to: skip, by the computer, the song, in responseto determining that the song is not appropriate; and tag, by thecomputer, the song.
 18. The computer system of claim 13, furthercomprising the program instructions executable to: skip, by thecomputer, the song, in response to determining that the facialexpression is not positive; and tag, by the computer, the song.